docs: Redis/Valkey cluster-mode migration spec with mechanized fencing model - #3756
Draft
tlongwell-block wants to merge 14 commits into
Draft
docs: Redis/Valkey cluster-mode migration spec with mechanized fencing model#3756tlongwell-block wants to merge 14 commits into
tlongwell-block wants to merge 14 commits into
Conversation
…g model Specifies the bb-public relay's migration from single-shard cluster-mode- disabled Valkey to a sharded cluster, in the house style of docs/git-on-object-storage.md and docs/multi-tenant-relay.md. The mechanized core is the fenced session directory key migration: the tunnel lease/generation pair must be renamed to share a hash slot, which is a live handoff of fencing authority under a rolling deploy. The TLA+ model (docs/spec/RedisClusterFencingMigration.tla) proves single-authority, generation-monotonicity, and one-way-door safety across the O->A->B-> backfill->C phase protocol; all five invariants are mutation-tested non-vacuous (5 mutants, each trips its intended invariant). The rest is gated engineering: split client architecture (deadpool cluster pool + separate RESP3 push_sender subscription client), sharded pub/sub as a sizing prerequisite (classic PUBLISH broadcasts to every node), the silently-partial SCAN in mesh discovery replaced by a scored-expiry index, and the staged ElastiCache mode change with cluster-mode=enabled named as irreversible. Inputs: Wren's deployed-source inventory and Dawn's migration-mechanics digs (channel buzz-redis-cluster-mode, thread 3440fed6). Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…ot-rules corrections, drain fix) Review blockers from Wren (three: E1 deployment-gate assumption, compatible- mode cross-slot legality unsupported, proof-surface overclaim) and Dawn (two: CROSSSLOT enforced on single-shard cluster-enabled nodes falsifies the A/B-in- compatible premise; renew=PEXPIRE makes any temporal drain gate unsound). Spec: A2/A3 rewritten (declared-keys slot check, undeclared-KEYS prohibition, slot rules from compatible onward as working assumption + pre-G3 probe); migration reordered A->B->backfill->drain->C entirely in disabled mode with compatible after C; C-gate now participation-based (B renew-migrates old-key leases; observed-zero, never wait-a-TTL); per-phase operation table for all six directory ops; proof scope stated honestly (write-side under E1); Deployment Gate section discharging E1 with rollback matrix and phase gauge; 1.5.0 receipts (verified free single-node); prior-art section (Stripe, GitLab, BullMQ, Sidekiq, Grafana, ioredis #1842) with locality contract and executable cross-slot validator in G2; failure-modes table extended. Model: mode variable disabled/compatible/enabled with slot rules from compatible; MigrateB (renew-migrate, same generation); RollbackPhase + DowngradePod with drain + reverse-max-merge guards; EnterCompatible/ RevertCompatible/EnableCluster doors; Inv_ClusterSafe -> Inv_SlotRulesSafe. A's generation dual-write removed: mutation testing proved it redundant. TLC green (12636 states, 3637 distinct); 9 mutants (M1-M9) all killed by their intended invariants. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Header still said A dual-writes the generation (removed after mutation testing); acquire-section comment lumped O in with the both-keyspace scripts (O touches only the old pair, itself cross-slot). Comments only; TLC re-run green, state count unchanged. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Review round 2 (Dawn) found the safety model could not distinguish the B2 fix from its absence: unguarded ExpireOld lets the C-gate be discharged by luck, so deleting MigrateB left all invariants green. Added DrainSpec/Probe_NotC — TLC from the stall state (full-B fleet, live old-key lease, no spontaneous expiry) with inverted verdict semantics: violation proves C reachable (93/44); mutant M10 (MigrateB removed = round-1 behavior) stays green, making the round-1 bug permanently visible. Main safety model unchanged (12636/3637, M1-M9 all killed). Also per Dawn N1: rollback matrix now names B->A as dropping every migrated session (A renews the old key; nobody renews the migrated lease; renewer treats Lost as fatal) — liveness cost, prefer forward-fix. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
TLC halts at the first violation, so generated/distinct counts on the drain probe's PASS run vary with worker count and scheduling (9/8 at -workers 1, ~100/~45 at -workers 4 — Dawn measured 114/50). Removed the count from the run transcript and noted only the verdict is the check; exhaustive-pass counts remain deterministic and cited. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
The drain-reachability probe (M10) guaranteed nothing mechanically: it was a documented procedure a reviewer had to remember to run. Deleting MigrateB leaves all five safety invariants green, so nothing in CI would have caught the round-1 bug coming back. scripts/check-tla-specs.sh runs both configs with opposite pass conditions: the safety model must complete with no violation, and the drain probe must report `Invariant Probe_NotC is violated` — a green drain run means phase C is unreachable from the stall state and fails the build. The inverted assertion greps the specific invariant name rather than checking TLC's exit status: a parse error or missing module also exits non-zero, and treating that as the expected violation would make the check useless exactly when the spec is broken. tla2tools is pinned by release tag and by sha256, since release assets can be re-uploaded. Verified in four directions on the committed script: unmodified specs exit 0; MigrateB stripped from DrainNext exits 1 via the green-drain branch; a spec that fails to parse exits 1 via the neither-verdict branch; a jar with the wrong checksum exits 1 before TLC runs. Runtime is ~3s including the jar download. The module allow-list is deliberately just RedisClusterFencingMigration, not a glob over docs/spec/*.cfg: MultiTenantRelay does not finish in a reasonable CI budget and GitOnObjectStore explores ~12.4M states, so attaching them to every PR would buy flakiness rather than safety. Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
| contents: read | ||
| steps: | ||
| - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 | ||
| - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 |
added 8 commits
July 30, 2026 13:36
…ecisions Dawn's G0 profile and P2 measurements (2026-07-30, receipts in RESEARCH/BB_PUBLIC_REDIS_G0_PROFILE.md and in-thread) landed three corrections to spec claims: - "per-topic slot spreading" overstated the spread unit: 87.8% of publish volume targets buzz:<community>:global (presence 60.8% + observer frames 27.0%), so the effective unit is per-community. Simulated hottest-shard imbalance ~1.5x fair share at 4 shards (p95) — D1 must not size by current/N. - P2 payoff is bounded by measured amplification (~5.3 pod-deliveries per publish on 15 pods), not worst-case broadcast. - D2 is now measured-closed (all pattern publishers lifecycle/admin-rate, ~10x margin), and D1 records its first inputs plus the honest caveat that per-family rate x latency explains under a quarter of engine CPU. Also replaces the stale ~400x figure with the fitted 9-day x12.4 and the rising per-op cost (1.18x/9d) as the P2 motivation. Model and configs untouched; doc only. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Dawn's P3 replica-subtraction measurement (2026-07-30, receipts in RESEARCH/BB_PUBLIC_REDIS_P3_ENGINE_CPU_DECOMPOSITION.md) resolves the open <25%-decomposition caveat and lands three spec updates: - D1 gets its gated input: ~40% of primary engine CPU is write-apply + publish propagation (the never-read replica carries 7.8pp of 19.5pp); ~60% is client-facing and divides only as connections/subscriptions redistribute. Notes the A4 dependency (propagation confines to the owning shard only after P2) plus two measurement cautions: the pub/sub-vs-keyed regression split is unidentifiable (r=0.997), and the apparent sublinearity is a ~1.1% fixed baseline. - Fallback row corrected: the replica is not idle — 7.8% is its own unsheddable load on a ~3.8d doubling, saturating alone in ~14 days. Kickoff's 'idles at 7%' framing removed. - Problem statement gains the measured fit window: sharp onset 2026-07-21 ~18:00Z; fits must start post-onset; curr_connections moves in pool-config plateaus, not demand. Onset unattributed pending deploy history. Summary timeline row updated to the fitted ~7d-to-80% with the degraded-fallback note. Model and configs untouched; doc only. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Dawn's delta review at 3292b35 caught the doc carrying two runways: the problem statement still held kickoff's eyeballed ~3.5-4d doubling / 8-10d saturation, while the summary row read the fitted ~7d-to-80% — and the eyeball figure was fitted across exactly the Jul-21-18:00Z regime boundary the adjacent fit-window bullet forbids. Problem statement now leads with the fitted post-onset clock (~3.4-3.5d doubling, ~7d to 80%, r2=0.92) and demotes the kickoff figures to a declared pre-onset eyeball. Same treatment for the ~150x/~400x family growth (fitted: x9.6-12.4 over 9 days). G0's gate description (line ~595) keeps its original tasking text deliberately. Model and configs untouched; doc only. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…way band, optionality framing Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…ml moved on main) Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
… 1.4d Dawn's Correction 2: stacked levers multiply M, days add — 1.882 x 1.330 x 1.111 = 2.782 -> 5.09d at Td=3.45d (4.93d interaction-aware). The ~3.4d figure required Td=2.55d, a fast-edge result reported as central. Also reconciles the kickoff-era r8g '~1.5 days' at line 90 to the fitted 1.4d used everywhere else. Conclusion unchanged: no infra lever outruns the curve. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Dawn's residual on 5baf0a1: the inline derivation used 1.33x, above the doc's own cited ~25-30% single-thread range — a kickoff eyeball rounded up past its ceiling and presented as derived. Restated as the vendor range 1.25-1.30x -> 1.1-1.3d, explicitly unbenchmarked. Wall sentence takes the optimistic edge (+1.3d) and restacks: M = 1.882 x 1.30 x 1.111 = 2.718 -> ~5.0d (interaction-aware 2.633 -> 4.8d). Conclusion unchanged: still ~4x short of the 21d likely migration. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
… a deploy Dawn's onset-attribution finding (RESEARCH/BB_PUBLIC_REDIS_ONSET_ATTRIBUTION.md): onset moves 18:00Z -> 16:40Z (15-min bins blurred it); attributed to organic community creation (58 -> 425 in 2h, same 5-min bin as the Redis breakout, flat subs-per-connection ratio excludes a code change). Community creation is now linear (~3,100/day) but Redis load stays exponential — intensification within communities — so the watch metric is channels-per-community. Adds the four-account cacheclusterid Datadog trap (must scope aws_account:433851229429). No planning number changes: 3.45d doubling and 5.2-9.0d band stand. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Formal spec + mechanized model for migrating the bb-public relay's Redis/Valkey usage to cluster mode, before the single engine core saturates (EngineCPUUtilization doubling ~every 3.5–4 days; ~8–10 days runway measured 2026-07-30).
docs/redis-cluster-mode.md— the spec, in the house style ofgit-on-object-storage.md/multi-tenant-relay.md: axioms, safety theorems, conformance gates, failure modes, implementation correspondence.docs/spec/RedisClusterFencingMigration.tla+.cfg— TLA+ model of the safety-critical core: live migration of the tunnel directory's fencing keys (lease + non-expiring generation counter) to hash-tagged names under a rolling deploy.Mechanized claims
TLC (with
-deadlock; the model has intended terminal states): no error, 4153 states / 1487 distinct. Invariants: single-authority (no two live leases per session across old+new keyspaces), generation monotonicity (no generation issued twice across the counter handoff), one-way-door safety (cluster mode enabled only after old-keyspace authority drained), plus TypeOK and lease-generation groundedness.All five invariant-bearing mutations trip their intended invariant (each run in isolation): dropped union checks (A and B variants), skipped max-merge (A and B variants), and a dropped phase gate into the final script version.
Key design facts encoded
cluster-mode=enabledis irreversible; compatible mode is the last revert point, and scaling/engine changes are blocked while in it.push_sendersubscription client (deadpool exposes no push hook — structural).SCANfails silently under cluster mode (random single-node routing) → replaced by a scored-expiry index with a completeness-asserting conformance test.rediscrate floor ≥ 1.4.1; dead-primary dispatch halt named as a residual failure mode on every released version.Provenance
Code pins at deployed commit
22be8bb35(imagesha-22be8bb). Inputs: Wren's deployed-source inventory and Dawn's migration-mechanics digs (channelbuzz-redis-cluster-mode). AWS facts verified viabb-public-operations-ro(auto-failover enabled, Multi-AZ enabled, Valkey 8.0.1).Draft until Tyler + Dawn + Wren review lands 9/10+.